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ABSTRACT 



An evaluation was conducted of a multinational virtual 
seminar held from January through March 1997 on delivering distance education 
using the latest technology. The seminar was organized by the University of 
Maryland University College (Maryland) and the University of Oldenburg 
(Germany) . Participants interacted with experts and, to a lesser-extent, with 
each other through electronic mail. They also participated in face-to-face 
seminars. Evaluation questionnaires completed by 41 participants (and 4 who 
were not enrolled but completed surveys) showed that all participants liked 
the seminar and most would participate again, if the price were $200 or less. 
The technical details of the seminar worked very well in most cases, and 
participants were not intimidated by the technical arrangements. The number 
of participants decreased throughout the seminar, which was not unexpected 
given that there were no firm expectations and deadlines. The seminar was 
found to provide top-quality professional development at a low price for many 
people. It proved that professional development in distance education can be 
carried out in a virtual seminar. (KC) 
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This is an evaluation report about a virtual seminar held from January through March 1997. 



The seminar had been organized by Dr Eugene Rubin, University of Maryland University College 
and Ulrich Bernath, Carl von. Ossietzky University of Oldenburg and had 45 participants from all 
over the world. This seminar for professional development had been sponsored by AT&T Foundation 
in the ICDE /AT&T framework of their „Global Learning Initiative The author was appointed as 
evaluator of this seminar - the views expressed in this article are the evaluators own views - he 
works at FernUniversitat Hagen , Germany, as manager of the ZIFF (Zentrales Institut fur Fern - 

studienfor s chung ) . 



Results of research must be of a kind that can be controlled by anyone else using the same data, 
methods have to be controlled and accepted as standard methods by the scientific community. The 
aim of research is intersubjective controllability and „objective“ data. The subjective views and 
presuppositions of the researcher must clearly be stated and discussed. The following report does 
not meet all these requirements in all cases- the author’s engagement in distance education over the 
last 25 years, personal views and acquaintances, wheather conditions of 1996/97, all might have 

intruded the original aims of the evaluation. 
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1 . The learning witness 

As long as schools have taught classes of learners, the notion existed that a student, 
while not interacting directly with the teacher might benefit in terms of learning from the 
interactions of others in the same setting. In modem elementary schooling group work 
(i.e. interaction among the learners without the constant control of a teacher) has grown 
in importance. 

Also in higher education student-student interaction has gained a respectable position in 
the discussion of possible settings. 

Now in virtual classes or virtual seminars this notion stays alive and we should expect 
that the proportion of student-student interaction is still gaining in importance. 



This situation represents the idea that learning can be organized to happen not only by 
direct interaction (dyads) of teacher and learner (even if a fellow-student is pushed into 
the role of a teacher) but also by indirect learning possibilities of the witness - lets call it 
„witness learning"- the witness being mute but awake and „intellectually“ following the 
other interactions in the setting. 



The question is now whether this setting can be transported also into the framework of 
distance education. There are many forms of distance education throughout the world, 
very few indeed will be of a kind where dyads of learning interaction will be excluded: on 
the contrary, almost all distance education institutions will have provisions for such 
dyads - mostly student-tutor interaction of some kind, sometimes for classical seminars 
or summer schools in order to make up for something that is thought of being missing: 
autonomous learning, in remote, asynchronous interaction. 



Theorists of Distance Education like Borje Holmberg, in claiming that the „proprium“ of 
DE be the unique one to one relationship 1 seem to insist on the incomparability of 
distance education with organized learning of any other kind. 



The big disadvantage of Holmbergs notion is that we still must calculate: DE is attractive 
to tax payers only if it promises the same results at less cost than the normal system. 



Lets calculate then: 



In a normal daily class - lets say 30 learners and a teacher - there will be a net-contact- 
time of interaction among teacher and individual student of less than 10 minutes a week. 

There is a ratio of 1:30 in direct one-to-one interaction, if it were not for the trust in 
witness-learning possibilities shools would have to close down. 

The one-to one interaction Holmberg promotes for DE means, that, out of organizational 
reasons, this ratio will normally not be possible: Holmberg speaks of a 1:1 ratio! This 
means while the tutor interacts with one student no other student is learning from this 
interaction. 




1 Holmberg, Boije: “The Evolution of the character and practice of distance education" 
in: Open Learning, June 1995, p 47-53. 
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In correspondence education there normally was nothing like „witness-leaming M . 

And, let me blunty state this, if every DE student was to build up a relationship with a 
paid tutor on this 1-1 basis, DE never would have worked: Successful tutorial conatct, 
by telephone, writing, e-mail etc. only functioned because the 1-1 relationsship will be 
asked for only by a very small proportion of students, the others either dropping out or 
learning successfully without a prescheduled one-to-one interaction; the question arises 
whether the successful DE institutions make a living on their drop-outs. 

There are many successful DE students never having asked for individual support and 
still keeping the pace and sitting the exam and passing it. This must have been even the 
bigger proportion and it seems to represent the didactic quality of the material presented. 
It is this group of sucessful students working with elaborated didactic material which 
makes me think that the proprium of distance education cannot be the one-to-one 
relationship Holmberg talks about. 



One-to-one relationship in DE is something we have to work for as a built-in-possibility, 
yet hoping that it never will become necessary for every student! 



When electronic discussions are offered, we find a new dimension in distance education: 
fellow-students might profit a lot again from witness learning just like in any traditional 
educational setting. 



So one of the evaluative questions for this seminar was, what participants think they have 
most learned from. We should expect that the learning meterial, in the case of the 
seminar the readings, would be the most important part and that own activities, such as 
communicative interaction either in messaging or in project work would be second: The 
results though coincide with the theoretical aspects just mentioned: Participants think 
that seminar readings on the web contributed most to their learning, but the second 
position is held by witnessing the message interactions - even more than actively 
participating in such interactions. 



Message-interactions like in the Hyper-News system of this seminar are as a technical 
possibility not new to distance education 2 : Theoretically this is a new dimension to 
traditional concepts of distance education: extensive use of mail, computers and even 
groupwise access to similar systems although known for years seem not to have been 
included properly into theory. We do have to re-think now about messaging systems on 
the internet as one of the more prominent structural elements of distance education: a lot 
of didactical work has to be done! 



2 • * 

Robin Mason, Anthony Kaye (eds): „Mindweave: communication, computers, and 

distance education”, Milton Keynes, 1988. 
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2. The Virtual Seminar 

Here is the introductory text for the virtual seminar: 



Welcome to the Virtual Seminar on Distance Education , a professional development activity for 
university faculty and administrators . This seminar was made possible by a grant from the AT&T 
Foundation , administered by the International Council for Distance Education (ICDE). 

In many ways , this seminar is an experiment. We are trying to accomplish several goals which we, the 
primary instructors, feel are critical to the development of university faculty who are emerging distance 
educators. 

We think that it is important for new distance faculty to have a conceptual and theoretical background 
to their practice. 

This will place their institutional models and teaching methods in perspective and they can learn from 
others who have preceeded them. 

It is useful for this to occur within a intemational/multicultural context, as the field of distance 
education is fast becoming a global activity. Distance education exists all over the world, and we can 
all stand to learn from the successes (and failures) of others. It is always tempting to assume that each 
of us n does it better " than others and that distance education is defined as what we do at each of our 
institutions. We all can benefit from having our minds expanded. 

We are aware that working university faculty and administrators have limited time available for their 
own professional development. They must fit their own training in among their other responsibilities. 
Participating in a distance seminar , not only allows faculty to engage in such training, it does it at a 
reasonable cost. Moreover, we hope that it will be a rewarding experience; an opportunity for faculty to 
feel they belong to the distance education community. 

Emerging distance educators rarely have the opportunity to interact with the master practitioners and 
theoreticians in the discipline. We looked at the virtual seminar as a way to arrange such interaction. 

We believe that all distance faculty and administrators are colleagues in a common pursuit. We 
therefore felt it was important that all participants in the seminar view each other and the n experts n that 
same way as colleagues who can learn from and inform each other. 

We are interested in developing a cost efficient method for the training of distance faculty worldwide. 
We want to develop a replicable system for faculty development which will attract new faculty, be 
viewed as of reasonable quality and substance, and which would be of a reasonable cost to the 
institution. 

We hope that the seminar fulfills these goals for you. 



? 
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With all of the above in mind , we will be engaging in some serious evaluation efforts in order to judge 
the attainment of these goals. So be prepared to be asked some questions at the beginning and end of 
the seminar and for those of you in Germany and Maryland (or nearby j, we will see you at a face-to- 
face evaluation session. 



How the Seminar will Work 

The seminar will be pretty much what was described in the brochure. If you haven't seen the brochure , 
take a look at it now here. (Please refer to the appendix of this text). 



The Schedule 

The seminar will be divided up into 10 sessions , each of which will last one week You will notice on the 
main page of the Virtual Seminar ; there are 10 weeks indicated. Each week will be ” hot " (that is, you 
can link to it) at the appropriate time. This may seem a bit " lock-step ” to some but there are reasons: 

1. Each of our ” experts ” are only required to be with us for their week They may stay longer, but are 
not required to do so. 

2. There are a number of topics to cover. We want to ensure we get through everything. 

3. Because of the time difference between our various participants , we need to ensure that everyone 
has an appropriate opportunity to interact. 



Thus, on any particular week you will be able to access that week's discussion, plus any previous week's 
discussions. But you will not be able to jump ahead until the appropriate week begins. You can always 
contribute more to previous week's discussions, but you will never be guaranteed that others will read 
them. Our goal is to maximize the relevant discussion on each topic. 



Here is how the schedule will work: (time table follows) 



Projects 

Each participant will be expected to do a project related to their involvement in distance education. 
These projects will be posted on the web site and a discussion group will be opened so that participants, 
instructors and experts can comment on them. We hope that this forum will be an opportunity for 
participants to get suggestions and help with issues related to their distance activities. Participants will 
be expected to e-mail their projects to the instructor for posting on the site. 
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Evaluation 

As we stated above , there will be several evaluation points in the course. You will be asked to complete 
a pre-seminar evaluation. This will be a form linked to from the main page and we will ask you to fill it 
out and submit it. It is multiple choice and easy to complete. 

You will also be asked to attend a face-to-face meeting in either Germany or Maryland. For those 
participants who do not live near either location , we will be sending a written questionnaire for their 
completion . THESE FINAL EVALUA HONS AREA REQUIREMENT OF ALL PARTICIPANTS IN THE 
SEMINAR! Your participation in these evaluations will help substantially in improving the seminar for 
future offerings. 

The German face-to-face evaluation meeting will be held on March 13-14 , 1997. 

The Maryland face-to-face evaluation meeting will be held on April 18, 1997. 



So what's next? 

The Virtual Seminar will begin on January 6th, 1997. 



Bookmarks: 

As soon as you come to the Website, be sure to place a bookmark in your browser so that you can easily 
return. We suggest that you bookmark the main page, as all links begin there. We would not recommend 
bookmarking the discussion groups as their addresses are complicated. 



Biographies : 

The first time you visit the Website, you will be expected to enter some biographical information to 
share with the other participants. All we are asking for (and all we want) are 300 words which describe 
yourself: your position, your discipline, your distance education experience, your distance education 
goals. We are confident that you can squeeze that into 300 words. We debated whether or not to have 
each biography in a separate file or all together as one file. If they were in separate files, download 
time would be quick, but the biographies would be a bother to print out. If they all are in one file, then 
download time is quite a bit longer, but it is very easy to print. We decided to put them all in one file. 
Photographs will be separate. If you want to see what a participant looks like, you will have to click on 
the photo link. 

Speaking of photos. if you haven't sent one in or put one up on your own site so that we can grab it, 

then do so IMMEDIATELY 1 You will find out for yourself that distance education often results in a 
psychological distance that needs to be overcome. We hope that photos will help. 
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Week One: 

On Monday , January 6th, the link to Week One will be active : C//dfc on that link and you will be asked 
to enter a Login and Password. These are the Logins and Password that have been sent to you 
separately by e-mail. When you enter each Week, you will see that the page consists of some written 
material or links to written material and some discussion. At the beginning of each week there will be 
very few comments, but as the day and week progress, there will be more and more comments . It will be 
easy to fall behind so that is why we have requested (at times depending on where you are) to come on- 
line on Mondays and Fridays, so that you can enter your comments and read the final summaries. We 
would also recommend that you come on-line several times during the week to check out the 
conversation, and respond to other's remarks. Leaving it all to Friday might be a mistake, and your 
involvement would be minimal. While ydu can make comments after Friday, most other participants will 
be going on to the next week and may not read them. We are leaving previous week's discussions 
available primarily so that participants can go back and reread and review, if they so wish. 

Further information and instructions will be given by the instructors within each week's discussion. 



The Conferencing System: 

The Virtual Seminar Conferencing System is HyperNews. It is a " threaded " conferencing system , which 
means that comments are shown in outline form so that one can see the "thread" of the conversation. 
You can also view all comments in a strictly linear fashion, one after the other. Be sure to read the 
HyperNews Instructions before starting and try to practice submitting a comment in that discussion. 
HyperNews Instructions are located at the bottom of the Main Page. 



Lost? Confused? Need help? 

Be sure to contact one of us if you have any of the following problems: 

Language difficulties (you don't understand what is written or said) 

Connection difficulties (you are having trouble reaching the website, the download is too slow, you 
forgot your password) 

Navigation difficulties (you don't know where something is located) 

Interaction difficulties (you don't know how to use the conferencing system) 

Life difficulties (you are having difficulty keeping up with the seminar) 



Good luck and welcome I 



O 

ERIC 



Ulrich Bemath 
Eugene Rubin 
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3 . Dealing with a virtual seminar 



Already the announcement of such a seminar created interest throughout the world - the 
problem with a net-based announcement is that only visitors of the site will learn about it, 
the problem with printed flyers to announce such a seminar is that only people who did 
have contact to the ones who distributed the leaflets might have known about the 
seminar. So one of the specialities in dissemination of the information was that the 
organizers or experts or involved members of affiliated teams were the communicators of 
such information. Already after a few days of application possibility the seminar was 
overbooked. So the organizers decided to go strictly by the first-come-first-serve basis 
which led to some irritations with colleagues and friends of the DE scenery. 



So one of the possible participants tried to get by the application procedure, calling up 
and asking why he as a professor should „apply“ to take part in such an interesting 
seminar, since he is well known internationally as being interested in DE. 



Cultural differences between Europe, where it might be wise to differentiate among 
possible participants and USA where the strict first-come, first-serve basis applies also to 
professors if they want to participate in such a seminar, became apparent. 

The list was closed with 45 participants from three groups (Marylanders, Germans and 
rest of the world) - a slight imbalance came out since the „rest of the world" group 
turned out to enclose Americans as well as Europeans living nextdoor to Germany. 

In the first weeks, before the seminar officially began, biographies from participants and 
their photos became next to the readings the prominent part of the seminar. 



Who were the participants? 
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3.1 F rom biographies 

Read this by omiting technique condensed version of participant’s biographies- 

I only copied expressions from the originals in the order of their appearance and I find it 
represents a kind of „course-personality“ 

We are currently developing our goals of web based education 
involved in research and future cooperation 
peer-reviewed expertise with return ticket. 

I have developed technology in creating a more meaningful learning experience, 
making new friends 

professorship is experience with Professor for to gain experience 
for the past 12 years working to improve the support, 
to orient faculty. Networked learning 

"drop out" time: I spent a year researching our children in embedded university setting, 
provided technology can best be used 
a bit specialized old Chinese 

Currently, I serve as Director of online scholarly discussion project. 

Design of optional discussion field research. 

I teach funded project distance education policy, 

distance education infrastructure, people I do not yet know, private enterprises. 

Now I have culture of learning. I have been working in integration of the Internet 
more than happy interacting with experts, empirical analysis in a peripheral way 
etc. 

I published bywork for a new design , research is focused 
DE practitioner, professional, competent for media, new learning paradigm 

real Virtual daughter, a novice in Interaction, appointed Executive 
internationally recognized children 

contract research in biochemistry of Education - semiotic features in multilateral 
committment 

megabytes of tutorials years later: I hope to be able, severe insomnia physically. 

This spring the Network will late in life video and other intensive session 
first-rate teaching co-ordinating challenging experience opportunity teaching 
profession 

director local organisers abuse researcher 

tired of traveling, gotten increasingly interested in any source both onsite and online 



3.2 Experts 

The experts of the virtual seminar belong to the world’s leading figures in Distance 
education. 

No wonder that the resonance to the announcement was so big: to have the possibility of 
direct contact with these experts seemed to have been one of the major incentives for 
participation. It certainly is a big experience and felt as an honour to be addressed by one 
of these people personally on a collegiate basis. The hope would be that there could be a 
kind of remembrance. It is a difference whether I can name some experts of the field or 
whether one of these renown experts remembers my name from previous contact. 

Net versatility of the experts was different by previous experience of virtual seminars and 
intensity of technology used previously. 

Both American experts are used to using the net - one of them being on a trip during his 
week , which shows that this kind of net communication is totally built in into the work 
schedule. 

The European experts seemed to be quite novel to the net experience: I know that Boije 
Holmberg shortly before the seminar began, connected to e-mail facilities and now is 
making intensive use of it. Otto Peters will catch up - or already has. What struck me 
when witnessing the week of Otto Peters was that he used paper and pencil in order to 
„design“ his answers to the participants before they were typed into a PC: lots of rubber- 
traces on these sheets of used paper give evidence of the elaborative kind of his answers: 
Yes, it is a difference in personal work-style, either to answer to questions as they come, 
sitting at the keyboard and typing rapid answers into the net or dealing with printouts of 
the questions and formulating word by word with rubber and pencil before typing it into 
the net. And there must be a difference seen from the participants angle, too. either 
typing in questions with the technique of dropping one key concept or the other and 
making the spring flow or formulating specific questions after extensively having read the 
sources. So one of the insights to be discussed later, that many a question had not been 
put into the net, can be explained from this dichotomy of work-style. 



3.3 Evaluation of virtual seminars 



For this seminar a new evaluative design had been promised. Normally the educational 
product is ready in its structure, design and applicability before the mass-production 
process itself starts. E.g. in DE programs a staff of experts decides what the curriculum 
will include. 

Producers have to stick to these decisions and learners have to consume what is 
presented. 

Experiential evaluation 3 should be 

• the term for a mixture of formative evaluation, where the production process still can 
be influenced after the first dissemination of the 'product' which stays open for 
changes 

• and an illuminative approach, where the life situation of participants, experts and 
organizers and the context of the product in use are recognized and reacted upon. 



Experiential evaluation is a process parallel to the first steps of design. In the first step 
the needed characteristics of the product, the idea of "what it is all about" has to be 
evaluated. 

Before the financing even is decided upon the rationale of the product must be described. 

The idea of experiential evaluation in the case of the virtual seminar was not to insist on 
high-quality performance in the process of developing distance education material but 
rather to insist on guidelines taken from practical needs of possible participants, more or 
less typical for a professional development seminar. 

There would be two ways to find out those needs: 

• One could develop a questionnaire and interview future participants or 

• develop a structure to be filled with a test product and evaluated parallel (at the best) 
to its first application or after its application and then revised. 



There have been three seminars with basically the same curriculum for graduate students 
starting in 1994 at Towson State University (in 1995 and 1996 as minimester seminars)- 
all three times Gene Rubin was guest at Uli Bernhards seminar. The blend of the 
readings had proven to be adequate and the idea to present the authors of the texts in a 
virtual seminar was only a consequence of the interest of these participants together 
with the software possibility to interact in the HyperNews system. 

The HyperNews system had been tested in a pre-workshop-event (Distance Education 
Development and Delivery on the WorldWideWeb) chaired be Gene Rubin and the 
annual IUC/UMS IDE ^Professional Development Workshop: Distance Education and 
Technology: Future Visions, May 6-8,1996 at College Park“ In this case the HyperNews 



3 Fritsch, Helmut, Experiential Evaluation: Study on Methodology of Evaluation in Open 
Learning. The conceptual framework of evaluation in a European LINGUA project 
http://www.femuni-hagen.de/ZIFF/experi.htm 



System proved to be adequate and finally lots of e-mail communication and personal 
meetings (e.g. at an international conference in Siegen ..Virtual Campus" sponsored by 
the Stifterverband der Deutschen Wissenschaft and organized by GMW, the German 
Association for Media in Science to which the author of this report served as vice 
president) preceded the start of the seminar. 



This process of experiential evaluation - an integral part of the production process itself - 
can be made clear by the FemUniversitat experience with developing vocabulary training 
programs during the last 10 years. 4 

Very early in the design of a vocabulary training program a very essential part of the 
software is installed: A so called 'log-file' will trace all actions of the user, record all 
mistakes, performances, times etc. In processing this 'log-file' one can produce long lists 
of mistakes, group these and in the next step analyse the 'common mistakes': Most of 
usage problems with vocabulary are typical for special native and target-language 
interferences, so we are in the lucky position to foresee empirically mistakes and to react 
upon these common mistakes by giving special comments, hints and help etc. So the 
learner sometimes even gets the impression of being in contact with a system using 
'artificial intelligence'. 

The longer this process of testing is lasting, the more participants make their mistakes: 
the more refined this training system will be developed. Diagnosis of typical mistakes is a 
very essential part of these products. Experiential evaluation can only be accomplished 
when a feedback-contact can be established and kept up. 

So I think the ideology of distance education, that distance education is not just the 
confrontation of a learner with highly adaptive teaching material („open learning") but 

that distance education starts with the communicative act after the confrontation 
with material, fits best with the concept of experiential evaluation. 

In the case of the virtual seminar this evaluation report might play a vital role for an 
ongoing revision process of such an initiative and could help to find out whether such a 
seminar should be offered again. But strictly speaking this evaluation cannot be anything 
but a summative report after the event of the virtual seminar. 

We have to cope with a process that is to be evaluated. It seems to be like in the history 
of modem art a kind of a ..happening" that we want to conserve. Yes, there are some 
500 pages which could be printed out, there are some hard server-data which can be 
reported, there are participants who might have changed their attitudes. There are also 
drop-outs. And there is the main question „was it worth the effort?" 



4 Kuflher, Helmuth: Das Programm COACH: Interaktives Vokabeltraining mit 
akustischer Sprachausgabe, in: Empirische Padagogik 1991,5 (Beiheft 2), 249-260 



3.3.1 Instruments for evaluation 
Personal communication 



One of these evenings in September, directly after a congress about „Virtual Campus" 
we met in good mood for dinner at my house in Fronsberg. Otto Peters and his wife, my 
wife Maria, Uli Bemath, Gene Rubin, and Desmond Keegan. It was a long evening with 
this phantastic pork dish with the sauce of figs and dates, sweet white cabbage and good 
French white wine. Already during the dinner an intensive academic discourse began. We 
made many allusions to past experiences, confronted ourselves with all the concepts of 
distance education, laughed a lot and decided that this evening would be one of the 
instances in life to remember: Originally we planned to have the participants of the 
seminar meet with their friends and cook that same meal, have that same wine and put 
their glasses after a toast to the virtual seminar on the CD-ROM which we had planned 
to document the whole seminar on. But the concept of conviviality realised on that 
occasion - the unity of leisure, labour, and learning - could not be kept up until the next 
morning: every one of us had his/her other duties but we agreed upon the importance of 
transcending virtuality into the real life. 



Questionnaires 

At an early stage of the project we decided that we would have several questionnaires - 
at least one in the beginning of the seminar and one in the middle and one at the end. We 
had these questionnaires developed and altered, using a type of questionnaire which had 
been developed by UMUC for a virtual conference in fall 1996. It had to be adapted and 
geared so that the data would flow from UMUC to FemUniversitat. To test this new 
instrument it was also sent to a group of former participants of a virtual colloquium at 
the ZFE of the FemUniversitat. 5 The results, though, are typical for distance education 
questionnaires: 20% answers- which is about the number of „active“ participants 
(although not identical) of that event: not enough for our interest to test the instrument. 
So we have decided to not make any cross-references. 

The technique of an online questionnaire was the only possibility to get the data needed 
in a very fast and uncomplicated manner: the only development work different from 
normal questionnaire development methods was the gearing of the results from the 
incoming e-mail (we used a system where out of the net the answer code was sent via e- 
mail to the FemUniversitat) into an Excel spreadsheet, where my eldest daughter, 
Miijam Fritsch, did a wonderful job. 

It was because of this technique, that one week after the beginning of the seminar the 
results of the opening questionnaire could be posted on the net. 



5 ZFE, Zentrum fur Femstudienentwicklung: Virtuelles Kolloquium: Medienentwicklung 
im Femstudium, Januar 1997 (only the printed articles of this colloquium, „to represent 
the state of the art“). - . 
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The second questionnaire was discussed among the author and the organizers, the results 
of the second questionnaire should represent the actual working situation of participants 
during the seminar. Time spent in the seminar, going to suggested URLs, etc. 

One of the project aims for my institution, ZIFF of the FemUniversitat, was to work on a 
possible standardization procedure for the evaluation of net-based seminars, so one of 
the outcomes of this report is the proposal of the development of a general questionnaire 
for web based seminars. 



On the other hand we have to report about something which only may happen with 
online questionnaires: We missed the inclusion of several gearing codes for the second 
questionnaire: So participants filled out the questionnaire and submitted it but a couple 
of answers were not geared to our site because of these missing codes: Result was that 
we had to ask those participants who already had answered to do part of the work again, 
so that we might get a consistent database. In a printed version of a questionnaire this 
cannot happen bacause proofreading procedures are standard: in an online questionnaire 
it will happen once, until testing procedures become standard, too. 



The final questionnaire was to test several didactical hypotheses - and it is still open how 
intensively we will be able to control all items for future use: There is, though, the 
extremely seldom found possibility to use interested participants in the seminar, who 
were turned down because it was „full“, as control group. But as long, as we are not able 
to tell these members of a control group whether there will be a similar offering of such a 
course in the future, we are hesitant to „use“ them. 



Interaction analysis 



Anyone who has the complete documentation of the messages (1301 KB) could analyse 
the structure of messaging according to different categories. The easiest category would 
be to list who is messaging to whom. The difficulty is that although HyperNews provides 
the possibility to refer the message to any previous one, most people take the last one to 
post their message. So we had to go into the message itself and look whether a content- 
reference was made or whether it was just the next message in the row. This 
phenomenon made one thing clear: Participants either referred personally to someone 
with a rather personal comment or message ( „I see you are from ... My home is near...“) 
or participants gave personal but „declarative“ statements with little reference to the 
previous message. 



Indeed this is one of the problems encountered in the virtual seminar: it is hard to qualify 
ones own message as either to be content-related to someone elses message or not. 
Because of this insecurity exactly where to put a message the communication process 
always will differ from a life discussion. 



There is the possibility to only look into the list of messages in order to find out, whether 
some new message had been posted to a theme or to any of the related messages: Since 
the icon „New“ did not change its colour in all cases when this list showed up, it was 
hard to differentiate whether a message signalled as being new had been read already 
before: This is one of the critical technical points to be mentioned about the HyperNews 
system. 




Text analysis 

Since all texts of interactions are available in archives there could be a deep analysis of 
the total communicative structure of the seminar. Statistical analysis of the interactions 
(who referred to whom about what) is carried through. The biggest problem, though, is 
not the analysis of such data but the qualification of each message-part in terms of 
communicative categories. Already at first sight many texts have a non-related, 
declarative structure or a strict personal communicative aim - but do not meet the 
requirements of a discourse, (to state what has been understood of the partner and to 
contribute in own terms to a common goal of insight). Three categories clearly can be 
found: 

• technical problems with the hope for help 

• declarative messages not directly connected to somebody elses message-content: 
sentences we would expect to find in the biographies but not in the messages. 

• messages of the discourse type, carrying on some arguments, taking position. 

So the question of relevance has to be posed - a question that is typical for all academic 
discussions online or in a group-setting face to face. 



Server statistics 

Gene Rubin has provided a vast number of pages called „server statistics" which show 
many different phenomena of this seminar 6 . The server where the course had been run 
gives detailed information on login-times, on KB retrieved, on weekly peaks of 
attendance, on every single site. There is the possibility to compare the „hard“ data of the 
server statistics with the questionnaire data and finally the interpretative data of other 
methods of inquiry and interpretation. One of the first insights of such statistics is that 
there are many participants who were extremely active but did not send messages as a 
result of their being logged in. The above mentioned theoretical element of the witness- 
learner is clearly proved by this observation: and there has been no other way of tracing 
this phenomenon except via the server statistics. Unfortunately we do not have pass- 
word-based statistics, so there must stay a rest of uncertainty in interpreting these data. 
Also we do have two separate files of server statistics, out of technical reasons the 
UMUC server gives separate statistics for UMUC members and for the rest of the world. 



6 Although it would be interesting to present all these statistics - they show lists and 
graphics of accesses, top documents by access count, top sites by access count, and top 
domains - data protection forbids the total documentation, but there is an example in the 
appendix. For future evaluations we should think about password-oriented evaluations 
on the server. 



Group-meetings 



Two group meetings had been scheduled: in order to have affordable cost for such 
meetings it was decided that one would have to take place in Maryland and one in a 
central place in Germany. The short chapter where these meetings are referred to must 
be seen as one of the core results of evaluation of this seminar because the net time for 
communicative acts of participants in these face to face seminars turned out to be much 
higher than throughout the seminar. 

One difficulty that is too late to be dealt with is that there has been a time lapse of almost 
4 weeks inbetween the face to face meetings - so that there is a possibility to misinterpret 
one of the final questionnaire items concerning this meeting in the cases where the final 
questionnaire had been turned in before the face to face meeting was held. 

Another phenomenon became clear during the interaction of the face-to-face seminars: 
Although it bothers the group-process we always have members of the group who 
interrupt, who speak up when it is not their turn and others who try to get into the 
discussion but hold back too long so that someone else will start. It’s hard to convey the 
notion of a list of speakers, and when finally it is someone’s turn, so many things have 
been mentioned inbetween, that again he/she will talk about: 

• either the issue having come to his/her mind the moment he/she raised the finger 
(probably having taken a note on paper) and by that missing the point where the 
discussion has arrived at, 

• or taking up the issue as it developed in the timespan and probably then to draw back 
because someone else already had put forward the same argument. 



3.3.2 Questionnaire-Hypotheses 

• A virtual seminar does not differ from a normal face-to-face seminar exept in locality. 

• Drop-out in a virtual seminar is higher (easier) than in a normal one. 

• A virtual cross-atlantic seminar cannot easily be built in into the everyday work. 

• The use of English as the only language results in an intensive cross-cultural contact. 

• Top rating of elements of the seminar will be reading, messaging, project-work, 
witnessing. 

• Participants will say that they reached individual cognitive goals. 

• Most participants would not pay $500,- for such a seminar. 

• The objectives of the seminar could have been reached by the readings only 



The database to process these hypotheses is far too small for any empirically sound 
procedure: that does not mean that this type of evaluation is useless, but, as indicated 
above, we may draw the most important conclusions also from such data even if only 
some 50% answered the final questionnaire in time. During the month of May we will, 
indeed, remind the other participants to fill out the final questionnaire, too. 
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The first printout of the results had been posted on the net the day before the face to face 
seminar in Maryland was held. So the participants of the face to face seminar already 
took a printout home. 



4 Results 



Unlike distance education courses we are not able to quantify results of participation in 
terms of learning objectives reached or not, since learning objectives had not been 
formulated in an operational way and there was no kind of pretest whatsoever. As 
expected, participation went down during the seminar as it will in all educational 
offerings which have no set of rules for achievement and no possibility to sanction 
discontinuation. The mass of insights for running such a seminar may well be on the side 
of the organizers rather than on the side of participants. 



The phase where technical difficulties occurred was more a European phenomenon and 
diminished during the seminar. Not all experts in the scenery are of the same routine to 
use the new media as the average participant. Cross-Atlantic contact was lower than 
expected. Critical comments from participants who attended the face to face meetings 
are few: almost unanimously participants would suggest to have such seminars again. 
The seminar seems to have reached its goals. From the evaluators point of view: 
unanimously participants say not only that they enjoyed the seminar but also that they 
would participate again. (96%) 



4. 1 Results of pre-seminar evaluation 
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The pre-seminar evaluation form was made accessible to the participants two weeks after 
the seminar began. 41 participants answered: In the first evaluation we presented a 
number of 45 participants having answered: After checking all the possible sources we 
have to cancel 4 participants who were not enrolled but answered the questionnaire; 
what we learn from this is: The questionnaires should remain protected just as the rest of 
the core of the seminar. What we did is we took these 4 people to strenghten the control 
group, so we have now more than a dozen of answering members (N=13)of the control 
group. One of the participants warned us when he found out that the submitting of the 
questionnaire was possible without password. 



Results of the 41 participants are shown below: 

1. how many seconds it took you from switching on your 
computer to come back to this spot (time in seconds)? 



2. How I heard about the conference: 

a. From the Brochure 

b. From the Organizers 

c. From a colleague 

d. Surfing the Web 

3. How I Use the Web (Check all that apply) 

a. I am teaching/ have taught a course on the Web. 

b. I use the Web to supplement classroom instruction. 

c. I plan to develop instructional materials for the Web. 

d. I am doing research related to Web learning. 



Mean: Median: 

147,49 130,00 



f 


% 


6 


14,63 


18 


43,90 


14 


34,15 


3 


7,32 


f 


% 


3 


7,32 


19 


46,34 


28 


68,29 


17 


41,46 
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e. I design Web instruction for others to teach. 

f. I administer programs that are using the Web. 

g. As a personal information resource. 

4. How much would you pay (U.S $) for this conference 

a. $50.00 

b. $100.00 

c. $200.00 

d. $300.00 

e. $400.00 

f. $500.00 

5. How fast is your connection to the Web? 

a. Very fast -T1 or better 

b. Pretty fast - ISDN or better 

c. Average - 28.8 modem 

d. Slow - 14.4 modem 

e. Very slow - 9600 modem or less 

6. Right now, is this session at the beginning, 

in the middle, towards the end of your working day? 

a. Beginning 

b. Middle 

c. End 

7. If you decided on the spot to leave work, do you have to 
ask, notify, or apply to anyone else? 

a. Ask someone for leave 

b. Notify someone 

c. Neither of the above - 1 just leave 

8. What % of text-files are you going to print-out locally? 

a. 0% 

b. 25% 

c. 50% 

d. 75% 

e. 100% 

9. What percentage of these files do you think will be 
printed out by the other participants? 

a. 0% 

b. 25% 

c. 50% 

d. 75% 

e. 100% 

10. Tell us the number of items in your bookmark-list 

a. None 

b. 1-10 

c. 11-20 

d. 21-25 

e. More than 25 

11. Do you ever turn off the loading of images? 

a. Yes 

b. No 



9 


21,95 


20 


48,78 


40 


97,56 


f 


% 


5 


12,20 


10 


24,39 


14 


34,15 


10 


24,39 


0 


0,00 


1 


2,44 


f 


% 


15 


36,59 


10 


24,39 


12 


29,27 


3 


7,32 


0 


0,00 


f 


% 


12 


29,27 


15 


36,59 


14 


34,15 


f 


% 


2 


4,88 


18 


43,90 


20 


48,78 


f 


% 


1 


2,44 


19 


46,34 


8 


19,51 


5 


12,20 


8 


19,51 


f 


% 


0 


0,00 


14 


34,15 


11 


26,83 


12 


29,27 


4 


9,76 


f 


0 


0,00 


6 


14,63 


7 


17,07 


2 


4,88 


24 


58,54 


f 


15 


36,59 


26 


63,41 
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12. How much time did you spend last week 
looking at a screen of a computer? 

a. No time 

b. 1-5 hours 

c. 6-10 hours 

d. 11-20 hours 

e. More than 20 hours 

13. Are you familiar with HTML ? 

Can you construct/change a web page yourself? 

a. Yes 

b. No 



f 


% 


0 


0,00 


1 


2,44 


7 


17,07 


13 


31,71 


20 


48,78 


f 


% 


24 


58,54 


17 


41,46 



Proposing a virtual seminar on the web would create uncertainties whether a minimum 
connectivity would be given. In order to find out about differences between net- 
connectivity from both sides of the Atlantic we chose to have an objective measure and 
asked the participants to shut down their computer and come back to the location of the 
questionnaire again: Surprisingsly there was not the expected difference between the 
Marylander group and all others: Marylanders have to wait 126 seconds to be back at the 
spot (median) and the rest 130 seconds. So we find almost equal net connectivity on 
either side of the Atlantic. 



More than two thirds of the participants give the reason for attendance as planning own 
instructional materials for the web. This is most important for a professional 
development" seminar. The price for such a seminar, according to the participants in the 
phase before it began, should not exceed $150. We will see how the response is after the 
seminar. 




Question No.7 was to determine the grade of autonomy during work: Indeed there is a 
small minority of participants who have to ask someone else for leave. So the grade of 
autonomy seems to coincide with the responsibility for ones work, typical for the 
teaching profession. 



Cutting the woods as an issue also in this seminar was appearent also at the beginning: 
There is a specific difference in answering question 8: How much are you going to print 
out? and the answer to question 9 :How much will others print out. We fin d only one 
participant who denies to print out anything- we will meet one again towards the end of 
the seminar, who explains that he tries to completely run an electronic workplace - at 
least for himself, and that he criticizes the hint of Eugene Rubin, that all in all it sums up 
to some „500 pages" material. 



We find 19% admitting to print out everything. But with a bad conscience, since only 
half of these think that other particpants will be doing the same. 

One of the participants wrote:“I use a high quality active matrix 480x640 LCD screen on 
a laptop and increasingly I find that reading from this screen an acceptable alternative to 
printing out text. I find the advantages of the laptop (I can read in bed, no flicker) 
outweigh the disadvantage of smaller screen size. I now use the same computer in the 
office instead of a desktop unit." 



Most participants did not print out directly from the net but saved the files somewhere 
for future reference, to print out instantly before reading seems to be the worst 
alternative: industry with these high velocity printers has its share in this development. 



22 
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Bookmarking is one of the features of net versatility: only one third of the participants 
are not - yet - of the type of versatility to use more than 25 bookmarks. 

Computer as the main working instrument seems to hold true for the majority - more 
than 20 hours a week is by no way typical for normal college/university teaching staff but 
it is for the participants of the virtual seminar. 

Two thirds of the group of participants are able to construct/change a Web page on their 
own: 

This has been the level of experience: ..professional development" really meant a seminar 
for people who use the net, who know it from the inside, who need what had been 
promised in this seminar for their daily work. 



4.2 Second questionnaire evaluation report 



During the second and third week of the seminar the second questionnaire was ..open" 
We find that 38 people answered the following questions. 

1. By what reaction of yours would people in your surrounding 

be able to tell that you are participating in this seminar? (Check all that apply) 

f % 
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a. Talking a lot about it 




22 


57,89 


b. Appearantly being exited 




7 


18,42 


c. Consuming more biscuits than usual. 


3 


7,89 


d. Skipping lunch. 




2 


5,26 


e. Screaming, shouting, complaining. 


0 


0,00 


f. Leaving late or coming early 




12 


31,58 


2. The net is most "connective" in my case 








from - to 


f 


% 


early morning 


(6:00 am - 8:00 am) 


13 


40,63 


morning 


(8:00 am - 10:00 am) 


22 


68,75 


late morning 


(10:00 am - 12:00 am) 


12 


37,50 


lunch 


(12:00 am - 2:00 pm) 


3 


9,38 


early afternoon 


(2:00 pm - 4:00 pm) 


3 


9,38 


afternoon 


(4:00 pm - 6:00 pm) 


3 


9,38 


late afternoon 


(6:00 pm - 8:00 pm) 


6 


18,75 


evening 


(8:00 pm - 10:00 pm) 


6 


18,75 


late evening 


(10:00 pm - 12:00 pm) 


7 


21,88 


night 


(12:00 pm - 6:00 am) 


7 


21,88 


3. How many hours have you spent with this seminar last week? 








f 


% 


a. Less than two 




7 


18,42 


b. Two to six 




23 


60,53 


c. Six to eight. 




6 


15,79 


d. More. 




2 


5,26 


4. Where is your personal frustration threshold when you want to connect to 


new item in this seminar? 












f 


% 


a. After 2 minutes. 




9 


23,68 


b. After 5 minutes. 




7 


18,42 


c. After 10 minutes. 




2 


5,26 


d. There is none, my PC is multi-tasking. 


18 


47,37 


. ^ ^ 


23 
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5. How many times did you enter the seminar last week? 







f 


% 


a. Once or twice. 




10 


26,32 


b. Every day. 




12 


31,58 


c. More than once a day. 




13 


34,21 


6. Right now, where are you? 












f 


% 


a. At my office desk. 




28 


73,68 


b. At home. 




10 


26,32 


c. Somewhere else in a motel or so: off duty! 




0 


0,00 


7. Did you make any printouts during the first two weeks? 










f 


% 


a.Yes, I did, less than 20 pages. 




17 


44,74 


b.Yes, I did, more than 20 pages. 




15 


39,47 


c. No, I didn’t. 




6 


15,79 


8. Please try to remember your PC situation the first two weeks of the seminar: 


how many hours your local system was not working correctly/ not available last 


week? 












f 


% 


a. 0 




20 


52,63 


b. 1-10 




16 


42,11 


c. 11-20 




0 


0,00 


d. 21-25 




0 


0,00 


e. More than 25 




1 


2,63 


9. Who is actually paying for the time and resources you use 


right now? 






f 


% 


a. I myself. 




7 


18,42 


b. My department / institution. 




29 


76,32 


10. During the first two weeks, did you read the messages? 










f 


% 


a. No, just saw the headings. 




0 


0,00 


b. 1-5 topics. 




4 


10,53 


c. Almost all. 




25 


65,79 


d. Definitely all of them. 




7 


18,42 


1 1. Would you go to other participants’ URLs whenever suggested? 








f 


% 


a. Yes. 




4 


10,53 


b. Sometimes. 




27 


71,05 


c. No. 




7 


18,42 


12. How much time did you invest in Week 


f? 










Mean 


Median 


(in minutes) 




234 


200 


13. How much time did you invest in reading the article for 


this week’s 


discussion? 










Mean 


Median 


(in minutes) 




70 


60 



O 

ERIC 



24 



21 



The fear that participants might be frustrated by rather technical difficulties of logging in, 
wandering around in the site, being sent to different URLs, with difficult connectivity at 
remote places, this fear found no confirmation: Not a single participant reacted with 
strong emotions while participating in the seminar: on the contrary, more than half of the 
respondents clicked the answer possibility that they talked a lot about their participating 
in this seminar. 



Net connectivity: We do have objective data concerning connectivity, but we neither 
know about the feeling it gives to wait for the connection (host contacted, waiting for 
reply) nor do we know, when exactly participants try to connect - or, give up at peak 
times. 



Now we know by the answers to question 2 that early morning hours are the favourite 
times to connect. The reason why this is very interesting is that time zones do not bother 
participnats: As long as there is no need to be on the spot at a fixed time this proves the 
assumption that participation is built in into the everyday work of participants. 



Only 25% work from home (question 6) and less than 20% have to pay the fees for 
connection themselves (in other words there are some lucky ones who connect from 
home but do not have to pay individually for it). 



Frustration threshold: Modem PCs are multitasking, so the connection can build up 
while you work on something else: Half of our group needs not to bother about such 
technical phenomena like world-wide-waiting: And the threshold for the rest of the 
group seems to be between 2 and 5 minutes. Since the average time to first connect to 
the site was some 2 minutes, there could not have been too much frustration in the case 
of this seminar. 



One frustration item must have been on the side of the participants, although they do not 
complain much about it: 40% report that their local system was out of order for a period 
of less than 10 hours; - one participant reports more than 25 hours for that week. 



Two thirds of the group came in every day, average time invested in the seminar was 
some 3 hours of which 1 hour was spent for reading the articles. 



Printouts were made by most participants, only 6 memebers of the group state that they 
didn't. 



4.3 Direct seminar Goettingen 

From March 13th to 14th members of the German group of participants met in 
Gottingen (a fairly central place to meet) to evaluate the experience of the virtual 
seminar. After a short introduction (who is here) several rounds of statements were 
initiated. Only 2-3 participants were known to each other before. 



A. Expectations 

In the first round we talked about the personal goals connected with the seminar. 
Flashlight - notes of expectations for this day were collected and discussed in dyads, so 
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that every participant would contribute to the discussion, the dyad dealing with notes 
from two other participants. 

By this method we assured that all notes were commented upon by at least two group 
members and the plenum of the group. 

Notes to the aims of the presence seminar (ungrouped) 

1) listen to critique, get an estimate of others, draw consequences 

2) language- and technical- barriers in the virtual seminar 

3) to get nearer to the other participants personnally 

4) discuss the schedule of the seminar (pressure of time and being online) 

5) to get to know the makers of the seminar 

6) to develop own project ideas for the German-speaking sphere 

7) to get an estimate on technical possibilities and hindrances 

8) Are virtual seminars good for learning? 

9) Is it good for communication, are participants content with it? 

10) Why did it go like it did: what were the influences? 

1 1) To be able to communicate more rapidly: without typing, reading, waiting 

12) to receive much more direct hints for practice in the seminar 

13) to learn something about the concept of evaluation of such seminars 

14) Ideally to start cooperation with other participants on definite projects 

15) To compare expectations and results 

16) to enlarge bi-and multilateral „tele-cooperation“ 

17) to collect proposals for improvement 

18) to encounter social phases in a virtual seminar (specific added value) 

19) to go deeper into the framework of DE 

20) to find new ideas on distance education 

21) to systematize the masses of infos on all topics 

22) get to know people in real life 

23) to participate because of an obligation 

24) exchange of experiences 

25) proposals for improvement 

26) Is virtuality good for special content areas? 

27) feasabilty of such seminars for daily practice? 

28) What would be the difference if we had met after the fourth week? 

29) Would it be an advantage, if we had met after a reading input nationally? 

30) the workplace Internet 

31) possibilities of group- work in WWW 

32) what will be the structure of such a seminar in 10 years from now? 

33) looking for synergetic effects with people from different backgrounds 

34) Try out a ..synchronous discussion" 

35) structures of online-communication 



(bold face printed items occured more than twice) 
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B. Didactical and technical insights 



1) The password had been given away in a couple of instances to team members etc (4 
cases: =9 more witnesses) But the messages only came from the participants 
themselves. It was suggested that the learning process could be intensified in terms 
of mute participants. The concept of „witness learning" - to do the assigned 
reading and intellectually follow the messages and discussions without participating 
actively - would seem to be an enrichment of the seminar because some of the 
participants found to be in that role themselves, at least partly. 

2) there is a special conviction that a direct seminar is much more intensive in terms of 
participation, number of communicative acts, emotion, and intellectual reward (in 
terms of time spent) than the parts of messaging via the net. 

3) Participants agreed to the statement, that the impact of the direct seminar is much 
higher (even in terms of time afforded) than the virtual parts - provided reading 
tasks were fulfilled. 

4) The structure of the comments in messaging was more declarative than 
communicative, (to be evaluated by qualitative methods of analysis of all 
communicative acts). 

5) A change of level and structure was felt to be irritating after the experts had left - it 
took a while until participants found their personal goals again- 

6) Net hours of seminar participation during the 10 weeks was 3.5 hours a week. 

7) majority of participants stated that they did not log in regularly 

8) a more topic-oriented structure was felt to be needed: the HyperNews structure was 
sometimes felt to be too linear. 

9) There was a conviction that there were lots of bye-communications (via e-mail 
among groups of participants) 

10) headings of the messages was for some participants a trigger 

11) the future structure of such a seminar should be first the readings, then a short direct 
seminar then the messages and the projects and after this another direct seminar. 

12) Discussion with the experts was definitely the highlight of the course 

13) Many messages have never been sent. 

14) The seminar was a good start into the professional field of DE. 

15) Discussion of the seminar in terms of ..teaching", ..learning" and ..monitoring" 
should take place, also the role of the organizers 

16) More or less the experience of the virtual parts of the seminar was felt as being 
teacher-centered. 

17) the offer of a chat group was noticed but not used. 

18) chat boxes and spontaneous comments should have an expiring date. 

19) How to transport the make of such a seminar into the DE structure with study- 
centers: Should there be compulsory online phases regionally or centrally? 

20) The outcome of the projects generated ..sitting ovations" 



21) Most participants feel the need for direct follow-up meetings in some national 
structure- there is a meeting (AGF) scheduled for Sept. 15/1 6th. All participants 
were happy that the site will be opened again July 7-1 1 . 



4.4 Face-to-face seminar UMUC 

Evaluation of the 2nd Evaluation seminar, Apr. 18th UMUC, College Park, Md. 

14 members of a group, of which 12 were active participants met on Friday, April 18th in 
the UMUC Conference Center to discuss the evaluation of the virtual seminar. The short 
overall impression of this face to face meeting resembled the Gottingen meeting even in 
small details. 

In both meetings one of the organizers and the evaluator were confronted with the 
group. In both meetings two scheduled participants were missing. Both groups found 
that it is worth while to start with synchronism in discussion. Also in the Maryland 
meeting members enjoyed the pace of discussion and the intensity of exchanging views. 

The steps of discussion were the same as in Gottingen: Participants were asked to write 
down their topics for group discussion as well as their personal goals for the day. 

After collecting the first set of notes (the second - personal - ones were for private 
evaluation towards the end of the day) these were discussed in pairs and then in the 
plenum. 

All in all we had three sessions that day - and spent about the same amount of time 
together as the Gottingen group did. The communicative style of the meeting was 
appreciated. The ratio of participants known to each other before was about the same in 
Gottingen and College Park. Average time spent with the seminar was about the same, 
weekly. 



Differences: 

• passwords have not been given away ( except in two cases for printing) 

• the twelve partcipants reported that they worked from some 30 differents sites (in 
Germany there were only 2 participants working from different sites, in USA only two 
worked from a single site) 

• the number of children reported was 28 (in Goettingen 19) 



Topics turned in for discussion: 

1) Expectation to leave with some concrete structures/configurations for a virtual 
course (structure and delivery and assessment) 

2) All was very useful, even though I was an active participant I had difficulty getting 
into theoretical discussion. 

3) How can such a seminar build up to foster and create real (vs. virtual) one -to-one 
and group interaction among ist participants. 



4) Where can we go from here - should there be a continuing list-serve, collabarative 
projects, what was the role of the leaders 

5) has to put in place distance ed. „structure“ policy etc. 

6) dealing with technical problems - students@home 

7) to learn more about practical aspects of web based instructions 

8) Discuss the pros and cons of the conferencing software we used in the seminar 

9) The look and feel of the interface software has a large effect on perceptions 

10) which did you learn more from: the assigned readings or the discussions 

11) strengths and weaknesses of the syllabi 

12) A „plan forward" for both seminars such as we participated in and collaborative 
efforts based on what we've learned 

13) The media appropriateness for the kind of material, content, learner 

14) Was theere enough support - technical, - human ? 

15) Was the level of the following appropriate: pacing, amount of material, size 
distribution, group-composition? 

16) differences in perceptions based on geography, experience in www, experience in 
DE 

17) was the www approach to learning appropriate to the kind of material,the kind of 
learners, the availability of technical and human resources? 

18) regional difTerrences? 

19) Where from here? 

20) First experience in working on a project with a person at a remote site via electronic 
media 

21) How about another course with some of the same and some new DE experts? 

22) 'How can you ensure and keep up full participation 

23) How can you identify and recruit members for a seminar like this 

24) Hands-on experience 

25) the asynchronous mode of communication makes it difficult to establish elements of 
decentered discussions 

26) the real student in the virtual setting not an alias 

27) What do the organizers believe to be the greatest things to do differently? 

28) What did the sponsors want from this, was it met? 

29) Did / will this seminar change the behaviour of participants in their workplace? 

30) Synchronous/asynchronous considerations 

31) Is ours a model (good/bad) of: introduction to DE. basic readings, technology, 
recruiting, evaluation process, online educ. etc? 

32) How to facilitate and maintain appropriate level of engagement without scheduled 
meeting? 



33) How directive should discussion be lead? 

34) Alternatives to weekly modules 

35) the degree of structure 



All these items were discussed, a new software (O'Riley WebBoard) was presented, 
open questions collected (is there research on chat-rooms?) and the hope was expressed 
that discoursive structures can be established and maintained, although „teaching faculty 
is like herding cats“. 

It was agreed upon that retention rate coincides with the grade of structure. 

The readability of messages and declarative vs. communicative style of messages as well 
as the role of teachers in such a seminar were discussed. 

A seminar like ours should not be maintained unlimited ( it is important to know when to 
go off stage); there must be a critical mass of sayers and readers, common agreement 
upon goals and moderation to ensure a „true academic discussion", which is not the 
same as the exchange of declarations. 



4.5 Communication analysis 

This is the most time consuming and frustrating analysis of the seminar because 
HyperNews( and most other News Systems) does allow to post messages to various 
topics at various times The idea was, to be able to comment on a message even a couple 
of days later, when the comment was rather directly connected to the content of the 
message - but normally people connected to the last message in the row). 

It was during the course of the seminar that subheadings were introduced so that a more 
content-related structure could be followed. But in the archives we not only find again all 
the wrong places where messages have been posted to but also during a time lapse new 
messages, posted in the following week- and probably not having been read by most 
participants. Also we do not find in the archives all messages compared to the printouts 
at the end of the active week (probably due to archiving at an early stage and not closing 
down the week instantly). 

Several dozens of URLs, the participants thought to be worth visiting, appeared. 

Small numbers of emoticons showed up in the first weeks. (I am glad not to have 
included the question for the favourite emoticon in the final questionnaire!) 

There are many possibilities to interpret the whole bulk of messages: the fairly easiest 
one seems to be the documentation of 

WHO adresses WHOM 

In almost all cases we found some relation to another message either by content, mostly 
by title and keyword. 

We did a soziogram-like listing of messages including the organizers and the experts. 

The table is rather wide but we can tell the following findings: 



z/ 



Thre are 115 messages sent in by the organizers - most of these messages were posted 
for structuring purposes but many rather intensive content-related messages are there, 
too. 

The experts were included in the messaging process more intensively than the 
participants (they sent and received more messages in one week than most participants 
sent and received in 10 weeks - with only few exceptions of very active participants). 

Boije Holmberg sent 31 messages, and received 25. 

Otto Peters sent 29 messages, and received 24. 

Gary Miller sent 24 messages, and received 12. 

Tony Bates sent 19 messages, and received 19. 

Participants wrote in the average 10 messages . 

Overall the participants received from other participants and individually from the 
experts 256 messages. 



Among the participants 127 messages (that is without experts and organizers) 
exchanged. 
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This part, the contact matrix will give evidence that Marylanders posted most (55), 
Germans next (42) and the rest least (30); There is a possibility to trace down the cross- 
atlantic communication - but not at this stage of evaluation. 



There are 8 members of the group having never received a message (except the ones 
meant for all the participants). One of the participants received 19 messages. 

We find the most active participation (if messaging were the only measure for 
participation which, as we know from face-to-face seminars, is not!) with 34 messages 
sent; there are 3 participants who did not post a single message, 3 more posted only one 
or two messages during the course of the whole seminar (N of messages sent <3 = 6). 



We know from the questionnaires that not everyone being silent was a drop out; on the 
contrary, from the Goettingen seminar we know that one of the most active participants 
very seldomly spoke up: and we can prove this by evaluating the server statistics, where 
this participant is indeed one of the most connective ones. 



The range of messaging is 704 messages in all 10 weeks. Week two being the most 
intensive one with some 135 messages. 



Two thirds of all messaging came from the participants, the projects not to be counted. 
In normal professional development seminars this ratio of participation in discussions 
would be excellent. 



We see here the importance of the technical system of posting - it is not for 
documentation reasons that we have to didactically design such a system, but out of 
reasons implicit to the choice of participants that messaging is the most important part of 
such a seminar. 



Indeed several participants put forward rather lenghty messages, articles like for a 
journal, for discussion in this expert group of distance education professionals. 

There also could have been a breakdown of the seminar after the external experts had 
gone: week 6 , the first one without expert already went down in numbers, decreased 
until week 10, where we witnessed a beautiful interaction: as a response to that simple 
question „Who is still with us?“ some thirty members responded, project reports were 
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announced at the site and in reading the documentation one can feel the relief that this 
endeavour had been worth the time. 



There is still the possibility for additional analyses of the communicative structures in this 
seminar but not in the framework of this first evaluation report. 



4.6 E-mailing 

I know that in the beginning and throughout the seminar there has been lots of e-mail 
contact not only among the organizers, the evaluator, the experts and the participants but 
also among participants as a group. I have not seen the e-mail folder of this project- I 
trust that what the organizers told me, that there had been days where the e-mailing 
outnumbered the messaging. Before Mid-May I received 82 pages of e-mail concerning 
the seminar and the contact with participants and the contact among the organizers.Also 
this aspect can only be treated in an additional study. 

Also project work was carried out via e-mail, we find in the seminar listing of projects a 
set of collaborative „papers“ which promise that the overall process of this virtual 
seminar will result in several joint initiatives. 



4.7 Final questionnaire 

Just in time for the end of the seminar we posted the final questionnaire - during the last 
week participants were asked to go to the main page of the seminar and fill in the 
questions. 

Technically we decided not to „gear“ the results via e-mail from UMUC to ZIFF but to 
put directly a link from the site of the seminar main page to the ZIFF, so we received the 
answers directly. 

The first printout of the final questionnaire could be distributed in time for the 
participants of the face-to-face seminar: the results also were put on the net under 

http://www.femuni-hagen.de/ZIFF/evarep3.htm 

When we tried to connect to this site out of the face-to-face seminar we had no 
connection: we later found out that just on that day there was some hardware routine 
work in our computing centre - exactly during the hours when in Germany (Friday night) 
connections would least be disturbed: not knowing that this was the time when we 
wanted to connect to the FemUni site from Maryland. 

Here are the results of the evaluation of the final questionnaire, answered by 28 
participants. 






When you joined the seminar, how important was it for you, to reach the 
following goals and which ones did you reach? 



l=most important, 3=not important 




1 




2 




3 


reached 


not reached 




f 


% 


f 


% 


f 


% 


f 


% 


f 


% 


Enlarge my knowledge about distance education. 


20 


71,43 


6 


21,43 


1 


3,57 


27 


96,43 


0 


0,00 


Learn about different systems of DE. 


14 


50,00 


11 


39,29 


3 


10,71 


25 


89,29 


3 


10,71 


Have direct contact to experts in DE. 


19 


67,86 


8 


28,57 


1 


3,57 


27 


96,43 


1 


3,57 


Experience a virtual seminar myself. 


27 


96,43 


1 


3,57 


0 


0,00 


28 


100,00 


0 


0,00 


Learn about the state of the art in DE-technology. 


17 


60,71 


11 


39,29 


0 


0,00 


22 


78,57 


5 


17,86 


Learn about how a virtual seminar is organised. 


21 


75,00 


6 


21,43 


1 


3,57 


24 


85,71 


4 


14,29 


Contribute to the field of DE-research. 


5 


17,86 


7 


25,00 


16 


57,14 


11 


39,29 


14 


50,00 


Get rid of my personal critical questions about DE. 


2 


7,14 


12 


42,86 


14 


50,00 


14 


50,00 


11 


39,29 


Get involved in the global perspective of DE. 


14 


50,00 


10 


35,71 


4 


14,29 


24 


85,71 


4 


14,29 


Get in touch with colleagues. 


12 


42,86 


16 


57,14 


0 


0,00 


27 


96,43 


1 


3,57 


Improve the level of my skills in DE. 


13 


46,43 


12 


42,86 


3 


10,71 


22 


78,57 


5 


17,86 


Acquire skills that I could use in the near future. 


20 


71,43 


5 


17,86 


3 


10,71 


25 


89,29 


2 


7,14 


I hoped to enrich my DE-related vocabulary. 


6 


21,43 


13 


46,43 


9 


32,14 


22 


78,57 


5 


17,86 



Do you agree with the following statements? 



I fully agree I partly agree I don't agree 



I think I reached the cognitive learning objectives of the seminar. 

Most of my fellow participants reached the cognitive learning objectives of 
the seminar. 

I would be able to formulate my own position on DE-theory. 

I use more elements of the jargon than before attending the seminar. 

I know my way around in DE-terminology. 

I had some language problems. 

To develop and deliver a virtual course seems easier to me now. 

I realised that "groups" emerged during the course of the seminar. 

I wanted to belong to a group. 

I belonged to a group. 

Social contacts with fellow participants are important to me. 

I think that photos and biographies are a general enrichment of any distance 
education course. 

At the beginning of the seminar I went through the biographies. 

I also referred to the biographies during the course of the seminar. 

I referred to the biographies especially when reading messages. 

I think that the communication during the se minar was not personal enough, 
in spite of the additional information supplied by the 
biographies/photos/homepages. 

I had direct email contact with fellow participants of the seminar. 

I plan to keep in touch with some participants of the seminar whom I didn't 
know before. 



f 


% 


15 


53,57 


6 


21,43 


13 


46,43 


9 


32,14 


13 


46,43 


1 


3,57 


11 


39,29 


10 


35,71 


12 


42,86 


7 


25,00 


18 


64,29 


24 


85,71 


20 


71,43 


11 


39,29 


6 


21,43 


4 


14,29 


10 


35,71 


17 


60,71 



f 


% 


13 


46,43 


19 


67,86 


12 


42,86 


14 


50,00 


13 


46,43 


10 


35,71 


13 


46,43 


14 


50,00 


9 


32,14 


8 


28,57 


8 


28,57 


4 


14,29 


7 


25,00 


12 


42,86 


9 


32,14 


16 


57,14 


8 


28,57 


9 


32,14 



f % 

0 0,00 

3 10,71 

3 10,71 

5 17,86 

2 7,14 

17 60,71 

4 14,29 

3 10,71 

7 25,00 

13 46,43 

2 7,14 

0 0,00 

1 3,57 

5 17,86 

13 46,43 

8 28,57 



10 35,71 

2 7,14 
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How much did the elements contribute to your personal success in the 
seminar? 







a lot 




a bit 




not 




f 


% 


f 


% 


f 


% 


The seminar readings delivered by mail. 


17 


60,71 


5 


17,86 


6 


21,43 


The seminar readings on the web. 


25 


89,29 


2 


7,14 


1 


3,57 


Additional recommended readings (not on the web). 


4 


14,29 


14 


50,00 


9 


32,14 


Recommended URLs. 


7 


25,00 


17 


60,71 


2 


7,14 


Actively participating in the message interaction with the experts and 
organizers. 


18 


64,29 


6 


21,43 


4 


14,29 


Actively participating in the message interaction with fellow participants. 


12 


42,86 


12 


42,86 


4 


14,29 


Witnessing the message interactions. 


20 


71,43 


8 


28,57 


0 


0,00 


Availability of biographies. 


9 


32,14 


17 


60,71 


1 


3,57 


Availability of a photo. 


4 


14,29 


18 


64,29 


6 


21,43 


Direct email contact. 


8 


28,57 


14 


50,00 


6 


21,43 


Conference Assistance. 


7 


25,00 


14 


50,00 


7 


25,00 


Seminar weekly archives. 


15 


53,57 


10 


35,71 


3 


10,71 


Chat group for the virtual seminar. 


1 


3,57 


8 


28,57 


16 


57,14 


Projects. 


9 


32,14 


10 


35,71 


9 


32,14 


Evaluation seminar. 


13 


46,43 


8 


28,57 


2 


7,14 



You subscribed to the virtual seminar and promised to participate. 

Did you? (Please check all that apply.) 

f % 
23 82,14 

8 28,57 

0 0,00 

1 3,57 

5 17,86 

5 17,86 



a. I participated all along, I wouldn't consider myself a dropout. 

b. My time schedule was too tight, sorry (my fault). 

c. I didn't want to continue, the seminar wasn't exactly what I expected (nobodys fault). 

d. I expected something different, you didn't meet my expectations (your fault). 

e. I was technically unable to connect regularly (technical fault). 

f. I wasn't able to participate throughout the seminar, for reasons out of my control (no fault). 



In drop-out theory we sometimes make a difference according to the level 
of engagement, where a possible drop-out happened. Please check the 
statement that fits best for you. 



a. Tdidnt even start the course (nonstarter). 

b. I probably belong to the "draw-backs", since I stopped the activities early. 

c. I dropped out after a regular start for a number of reasons. 

d. I was quite active throughout but stopped the activities before the official end, so Pm a "no-show". 

e. Pm not a drop-out. I participated actively throughout the whole seminar. 

f. Pm not a drop-out I was present throughout the whole seminar, people might just not have noticed me. 



f % 

0 0,00 

0 0,00 

1 3,57 

1 3,57 

14 50,00 

10 35,71 
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What do you think should happen with the seminar, after it is officially 
finished? 



a. It can safely be closed 

b. It should stay open for the participants for at least another few weeks. 

c. It should partly be opened to the public: anybody should be allowed to access the readings. 

d. It should be opened, so that anybody can access the readings and the discussions. 

e. It should be completely accessible but unmoderated, so that anybody can send in new comments. 

f. It was great and should stay open and moderated as long as possible. 

g. I dont really care. 



f % 

0 0,00 

16 57,14 

6 21,43 

3 10,71 

0 0,00 

5 17,86 

0 0,00 



Final Questions 



Did you enjoy participating in the virtual seminar? 

Would you participate again in a virtual seminar like ours? 



In case our virtual seminar would be offered again, would you recommend it? 
Would you recommend it, even at the price of 1,500 US$? 





Yes 




No 


f 


% 


f 


% 


27 


96,43 


0 


0,00 


27 


96,43 


0 


0,00 


26 


92,86 


1 


3,57 


10 


35,71 


15 


53,57 
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5. Conclusion 



Organizing such an international event, participating in many content-related interactions 
with the experts and the participants, e-mailing back and forth, structuring the flow of 
the topics, cross-linking participants with special questions, participating in the face-to- 
face seminars, beeing in good mood all over the 10 weeks of the seminar, working day 
and night, travelling half the globe and being responsible for the site which hickupped 
only once- and received another backup-site in Oldenburg, dealing with the external 
experts, ckecking the expenses and doing all of it in much less than one year: 
Participants, who now can imagine how much work it is to run a virtual seminar, would 
certainly be glad to attend such a seminar again. And even at the price of the overall 
budget of $ 80.000 the conclusion of the evaluator is: 

You will not get such quality professional development at such a low price for so many 
people again. The reason is that both organizers have their other duties and that this 
seminar was the first event of that kind with a lot of additional personal input which 
cannot be paid for. So Thanks to Gene Rubin and Uli Bemath we now know how 
professional development in Distance Education could be organized and carried through. 



Hagen, May 15 th , 1997 
Helmut Fritsch 



Web Server Statistics 

Week of 01/19/97 to 01/25/97 



Totals 



Item 


Accesses 


Bytes 


Overall Hits 


1,153 


11,685,295 


Home Page Accesses 


0 


0 



J 

Unique sites served: 108 



Unique documents served: 96 



Accesses per Hour 



Figures are averages for that hour on a typical day. 




00:00 03:00 06:00 09:00 12:00 15:00 18:00 21:00 
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\mk "S' 



Hour 


Accesses 


Bytes 


Bits/Sec 


Bytes/Sec 


00:00 


4.71 


99985.71 


222.19 


27.77 


01:00 


3.86 


9094.14 


20.21 


2.53 


02:00 


8.71 


63344.86 


140.77 


17.60 


03:00 


12.71 


200476.29 


445.50 


55.69 


04:00 


15.14 


114918.57 


255.37 


31.92 


05:00 


14.43 


138846.86 


308.55 


38.57 


06:00 


11.14 


93920.43 


208.71 


26.09 


07:00 


9.43 


117831.00 


261.85 


32.73 


08:00 


10.14 


108247.29 


240.55 


30.07 


09:00 


16.57 


157623.00 


350.27 


43.78 


10:00 


9.71 


91984.29 


204.41 


25.55 


11:00 


8.43 


67237.14 


149.42 


18.68 


12:00 


9.00 


94628.43 


210.29 


26.29 


13:00 


1.29 


14887.71 


33.08 


4.14 


14:00 


3.29 


33124.43 


73.61 


9.20 


15:00 


3.86 


41269.00 


91.71 


11.46 


16:00 


4.14 


29524.57 


65.61 


8.20 


17:00 


3.29 


32992.57 


73.32 


9.16 


18:00 


0.71 


11190.29 


24.87 


3.11 


19:00 


2.43 


40323.00 


89.61 


11.20 


20:00 


2.43 


31637.00 


70.30 


8.79 


21:00 


3.86 


31589.14 


70.20 


8.77 


22:00 


3.71 


3186.43 


7.08 


0.89 


23:00 


1.71 


41465.71 


92.15 


11.52 



Top 50 Documents by Access Count 




ERIC 



Rank 


URL 


Accesses 


Bytes 


1 


/ide/seminar/ban-conf. eif 


340 


204,750 


2 


/ide/seminar/hr.eif 


323 


3,692 


J 


/ide/seminar/bios/millerl .eif 


63 


2,857,284 
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